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introduction 



Dirsctors of continuing eujcation programs 
often express concern about evaiuating the im- 
pact of their programs. Funding agencies, pro- 
gram participants, and other interested persons 
share this concern, in settings where research 
experts are avaiiabie to participate in evaiua- 
tion, the types of methods used and knowiedge 
gained, such as questionnaires for measure- 
nient of attitude change, sometimes have iittie 
direct utiiization in changing the program. 
Research experts share the concern for effec- 
tive usefui program evaiuation. 

One probiem reiated to evaiuation is that train- 
ing program directors and facuity often do not 
gather the kinds of information which couid 
make them aware of new possibiiities for im- 
proving their programs. They are sometimes 
unaware of information that couid be fairiy 
easiiy obtained and that could have a consid- 
erable impact on their programs, it has also 
been noted that a constant frustration for pro- 
gram directors is the great limitation of time 
available for administration, including program 
planning and evaiuation. 

These Guidelines have been written with the 
hope that they will be useful to training pro- 
gram directors and to research persons who 
consult and participate in program evaluation. 
The Guidelines can be an aid to such persons 
as they explore the range of information useful 
for planning, designing and evaluating their 
programs. 

The emphasis of these Guidelines is on the 
types of information that provide a basis for 



program administrators to make constructive 
changes in their programs. Evaluation data not 
only includes information more customarily 
thought of in relation to assessment of educa- 
tional methods and outcome, but information 
essential for program planning as well. 

in order to develop the Guidelines an Ad Hoc 
Advisory Committee on Continuing Education 
Program Evaluation was convened in April, 
1969, sponsored by the Continuing Education 
Branch, Division of Manpower and Training 
Programs, National Institute of Mental Health. 
Fifteen experts representing a variety of disci- 
plines from universities, training organizations, 
and government agencies in addition to ten 
NiMH staff members met and discussed the 
many facets of evaluation. 

The Guidelines are necessarily broad and de- 
signed so that training program directors may 
select and use the items which are most rele- 
vant to their programs. The Guidelines are not 
designed as a program planning guide, except 
for the evaluation aspects of the program. How- 
ever, the principles discussed may be helpful in 
developing more comprehensive plans based on 
principles of (1) community involvement, (2) 
education, (3) administration and finance, and 
(4) the disciplines(s) being taught. 

It is anticipated that this initial edition of the 
Guidelines will later be revised on the basis of 
feedback from persons who have used them. 
Suggestions as to which portions should be re- 
tained, deleted, modified, illustrated or supple- 
mented will be most welcome. 
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what is educational evaluation 



Evaluation and Program Objectives 

Well defined program objectives are crucial for 
purposes of program evaluation. The objectives 
may be defined in terms of criteria of mental 
health for the community or population being 
served, in terrns of changes in the learners or 
their organization.?, in terms of secondary phe- 
nomena which the educational program is ex- 
pected to induce via the learners, in terms of 
increments in demonstrable knowledge and 
skills or in terms of demonstrable effectiveness 
or the relative merits of one educational meth- 
od compared to snother. 

Determination of objectives is a technical proc- 
ess in itself, and much has been written on this 
subject in literature on administration, research 
and education. For any continuing education 
program the objectives should be determined 
for a variety of levels, not all of which are 
necessarily incorporated into formal evaluation 
of outcome. 

1. General institutional objectives. 

2. Educational program objectives. 

3. Educational activity objectives. 

First, the sponsoring institution(s) should have 
a representative advisory committee which in- 
cludes administrators, faculty, participant train- 
ees and consumers of services. General institu- 
tional objectives should be clarified in terms of 
the uniqueness of the sponsoring organiza- 
tiori(s) and their mission(s), in terms of the 
available manpower and in terms of the ulti- 
mate population being served. This may re- 
quire considerable discussion and clarification 
based upon available data as well as philosophy 
and values. Since every organization has limi- 
tations in resources for accomplishing its pur- 
poses, the objectives should be translated into 
priorities and strategies to assure that the lim- 
ited resources are employed in a feasible man- 
ner for maximal impact on high priority needs. 
Such a strategy naturally includes concepts of 
long renjje development as well as immediate 
impact, concepts of learning and change in 
individuals and organizational teams, concepts 
of spread effect and secondary impact via key 
catalysts within the system and concepts for 
initiating self-perpetuating processes based 
upon the motivation of persons in the system 
rather than leaving the' program overly depen- 
dent upon thG motivation of the initial planners 
and administrators. 



Within this context and consistent with these 
general objectives, tfie educational program ob- 
jectives should be determined in terms of what 
types of changes in knowledge, skills and be- 
havior are expected in what types of personnel. 
The participants in the continuing education 
program are thus selected on the basis of their 
priority significance for accomplishing the gen- 
eral objectives of the sponsoring orqaniza- 
tion(s). 

Within this context more specific educational 
activity objectives can be determined for any 
specific course or other educational activity. 
Educational objectives are preferably defined 
in terms of measurable behavioral change, 
which can thus provide a basis foi evaluation 
of results. Educational methods are preferably 
selected only after the educational objectives 
have been determined, and the methods should 
be consistent with the objectives. For example, 
changes in skill are not usually obtained by 
lecture methods. Evaluation of methods is usu- 
ally intermediate or secondary to evaluation of 
educational outcome or achievement of educa- 
tional objectives. By the same token, evalua- 
tion of a specific course is secondary to the 
achievement of overall program objectives. 

Regardless of which aspects of a total program 
are selected for the most intensive evaluation, 
there should be an internal consistency in dif- 
ferent levels of objectives and methods, and 
some type of evaluation is necessary at all 
levels. 

In summary, the most useful evaluation is 
closely associated with “keeping your eye on 
the ball" rather than oeing caught up in inter- 
mediate or esoteric facets of the program. 
Clearly defined objectives and related evalua- 
tion of continuing education programs also en- 
able the funding agency and other sponsoring 
organizations to attain their objectives, which 
in turn contributes to the improvement of high 
priority mental health services at all levels. 

Educational Evaluation— A Circular Process 

For the purpose of these guidelines, educa- 
tional evaluation is defined as the process by 
which relevant information is constantly gath- 
ered and fed back into the program to be 
used as the basis for enlightened decision 
making leading to improvements in the pro- 
gram. 
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Evaluation has an important function in well- 
planned continuing ed jcation programs from 
their earliest inception, sunh as helping to de- 
termine the continuing education needs in a 
community. 

Evaluation is essential in each of the following 
stages of program development; 

1. Assessing the mental health problems of 
the community and the specific needs they 
reflect for continuing education. Depend- 
ing upon the nature of the sponsoring or- 
ganization, the “community" may be local. 
State, regional or national. 

2. Defining the educational objectives in spe- 
cific behavioral terms. 

3. Defining the target audience for the pro- 
gram, and developing methods for selec- 
tion of the desired trainees. 

4. Determining the educational content and 
methods to be used. 

5. Developing plans for gathering evaluation 
data and feeding relevant results back into 
the program to accomplish constructive 
change. 

Educational evaluation for continuing education 
is thus conceived as a continuous, circular 
process. The cumulative evaluation at the end 
of one year of a program should provide a basis 
for the planning and evaluation process during 
the second year. This continuous, circular proc- 
ess helps integrate evaluation into program 
planning and implementation. Assessment of 
the outcome and effectiveness of a specific 
educational activity or program for a particular 
year is only one phase in the total evaluation 
process. 

Educational Evaluation and Educational 
Res&arch 

Evaluation and research have many similarities. 
In a broad sense research includes evaluation, 
and evaluation may use research procedures 
and fact finding methods. However, educational 
evaluation and educational research differ in 
several significant dimensions. 

The purpose of research is to discover and ver- 
ify principles and to develop new knowledge, 
which may or may not be immediately applied. 
The purpose of evaluation is to provide infor- 
mation for enlightened decision making and 
application to improve the ongoing program. 

Elaborate methods are generally devised to 
make it possible for scientific investigators to 
divorce themselves as much as possible from 
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value judgments. Findings in specific basic re- 
search projects are not ordinarily fed back into 
the project to accomplish immediate change. 
Rather, the program is kept constant for the 
period of the research so that one or more ele- 
ments of induced, controlled change can be 
precisely measured. 

Research principles and methods are often 
used in evaluation. However, the capacity of 
a continuing education program to apply 
highly sophisticated educational research meth- 
ods in their program evaluation will depend 
upon the resources available. Most continuing 
education programs have neither the need nor 
the capacity to utilize highly technical research 
methods, let alone perform basic research. 

Educational program administrators view both 
the methods and the purpose of evaluation 
from a wide variety of perspectives. These 
range from views that equate evaluation with 
formal research procedures to very informal 
views of evaluation that are more akin to the 
opinions, judgments and intuition of adminis- 
trators. 

Naturally, in a long range perspective research 
program development operates by much the 
same principles as educational or service pro- 
gram development. Basic principles of program 
evaluation and program development apply to 
all. The above distinctions between research 
and evaluation are valid in the context of these 
Guidelines and are too often overlooked in pro- 
gram evaluation. However, this discussion is 
not intended to overlook the Importance of 
value judgments, application of findings and 
program development In research work. By the 
same token, an educational program will at 
times hold certain variables constant and keep 
value judgments to a minimum in the interests 
of evaluation. 

Other Dimensions of Evaluation 

Evaluation may be regarded in several dimen- 
sions, such as (1) ultimate vs immediate vs 
intermediate outcome (2) broad vs focused im- 
pact (3) long range vs short term (4) process vs 
outcome and (5) cost effectiveness vs educa- 
tional effectiveness. 

In terms of ultimate objectives each program 
can be judged for its contribution in the effort 
to solve mental health problems. In terms of 
intermediate objectives a program may be eval- 
uated for its effect on the functioning of the 
learners. How did the educational program 
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change them, and how will this in turn affect 
mental health in their communities? In terms 
of immediate objectives a program may be 
evaluated for the effectiveness of the educa- 
tional methods in fostering learning and be- 
havioral change in the learners. 

Assessment of ultimate objectives, that is 
changes in the population served by the train- 
ees, is highly desirable but can be difficult and 
expensive. When expert resources are available 
for such things as epidemiologic studies these 
may provide a means for assessing the effec- 
tiveness of the continuing education program 
in terms of improved indices of mental health. 
Such evaluation would usually be accomplished 
by integrating the educational program within 
existing programs for planning and delivery of 
services, as compared to supporting such eval- 
uation as part of the educational program. 

Another dimension of evaluation is the diffi- 
culty assessing broadly dispersed ripple effects 
as compared to focused impact of the educa- 
tional program. 

Often educational program administrators 
would see the ultimate outcome on the mental 
health of a population as too broad and indef- 
inite for assessment compared to their focus 
on the intermediate outcome of changes in 
trainees or immediate satisfaction with teaching 
methods. However, ultimate vs intermediate 
objectives need not be synonymous with 
breadth vs fine focus. Selection of a limited 
specific criterion of change in a limited specific 
population can make the assessment of both 
ultimate and intermediate impact an easier 
task. For example, a measurable ultimate objec- 
tive might be the reduction in rate of arrests 
for alcoholism in a population of 200,000. In 
this context the design and evaluation of impact 
of a continuing education program (impact on 
both trainees and the population at risk) could 
be more focused and feasible than assessing 
the impact on 200 participants of a weekend 
symposium on alcoholism, let alone assessing 
the impact on their clients. Naturally, not all 
continuing education programs can or should 
provide such a fine focus on the ultimate objec- 
tives. However, problems of educational evalu- 
ation are compounded by the tendency for edu- 
cators to focus on immediate and intermediate 
objectives— or on "satisfaction” rather than any 
more specific educational objective. 

tvaluation of long range oOiectiveo, such as a 
five year program, is obviously a different task 
than evaluation of short term objectives, such 



as the effects of a specific course. Program 
development may remain static or deteriorate 
if evaluation is concentrated only on short term 
objectives. Not infrequently a "successful" con- 
tinuing education project in the first two years 
can flounder on “recruitment problems" by the 
fifth year. 

Evaluation of educational methods and process 
cannot remain divorced from educational out- 
come. A continuing education program may 
rely so heavily on a process of demonstrated 
value, such as supervised clinical work or small 
group methods, that the program becomes 
characterized by “riding a hobby horse". Ulti- 
mate and intermediate objectives may be lost 
from view, only a special self-selected group of 
trainees may be reached, and the satisfaction 
of faculty and trainees with the process may 
obscure an abortion in program development. 
On the other hand, evaluation of outcome in 
terms of ultimate and intermediate objectives 
may be an empty exercise if the educational 
process and methods are demonstrably unsatis- 
factory. "Happiness data" is not a substitute 
for assessment of outcome, but student satis- 
faction is obviously an important ingredient 
for the educational process. In a related di- 
mension, evaluation of methods and outcome 
should include assessment of efficiency as 
well as effectiveness of the methods and 
process. 

The director of an educational program cannot 
be oblivious of cost effectiveness, no matter 
how convincing the evidence of educational ef- 
fectiveness. Generally speaking, the more costly 
the education per trainee-hour (or per unit of 
assessed outcome) the greater the need to 
demonstrate the spread effect or ultimate 
impact of the program. High cost may con- 
ceivably be justified by virtue of assured 
secondary influences or activities of the key 
persons being trained or by virtue of initiating 
a self-sustaining process that requires rela- 
tively small cost for the ultimate outcome. 
Cost effectiveness thus becomes related to 
motivation of trainees and their employers, 
their willingness to contribute to financial 
support of the educational program and their 
potential for accomplishing the ultimate ob- 
jectives of the program. Assessment of cost 
effectiveness is important to program planners 
to assure optimal utilization of limited re- 
sources. It is also important for attracting 
further support from trustees and other fund- 
ing resources. 
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Evaluation and Vested Interests 
Any continuing education program can and 
probably will be evaluated from a variety of 
points of view. A well-constructed evaluation 
plan will make provisions for obtaining feed- 
back from as many of the following as possible; 

1. The Community: How does the program 
help the total community? Does it fit into the 
community health priorities picture? Does it fit 
into the community organization and leadership 
situation? Have the trainees been selected on 
the basis of their strategic impact for the com- 
munity? How well does the program fit with 
activities of related and/or conflicting interest 
groups? What changes have occurred in the 
community and the target population as a re- 
sult of the continuing education program? 

2. The Trainees: Do the overall training goals 
and objectives of the program meet the needs 
of the target population? How valuable v/as the 
training experience to the trainee group? Did 
the program contribute any fringe benefits such 
as career advancement, academic credit, en- 
hancement of role? 

3. The Sponsoring Organization: How well 
does the program fit into the structure and 
tribute to the identity of the training institu- 
tion? How does it contribute to the aims and 
services provided by the institution? Does the 
program involve organizational changes which 
would enhance the institution’s capacity for 
achieving its objectives? 

4. The Program Director and Staff: How do 
they see their roles in the project? Are their 
activities viewed as professionally helpful and 
appropriate? How do their activities affect their 
careers? Do their roles carry high or low 
status? 

5. The Funding Organization: Does the appli- 
cation reflect a carefully planned program 
based on need? Is the program feasible? Is the 
program strategic in its impact for purposes of 
accomplishing the objectives of the funding 
organization? Are potential participants in- 
volved in program planning? Do plans call for 
efficient administration and financial effective- 
ness? Is the program integrated into other com- 
munity efforts relating to the same problem? 
Are educational objectives identified? Are pro- 
visions made for effective instruction? Is the 
subject matter relevant and adequate in quality 
as well as quantity? Does the program design 
incorporate evaluation based on the objectives 
of the planned training program? Are the ac- 
complishments consistent with the priorities of 
the funding organization? Are the administra- 



tive policies and procedures consistent with 
those of the funding organization? Will the pro- 
gram assessment provide data appropriate for 
progress reports and/or final reports? 

With the possible exception of the sponsoring 
institution, the funding agency, the training 
director, and the trainees, very few individuals 
in the opinion-forming groups described in the 
previous section will be asked to make formal 
judgments about a continuing education pro- 
gram which comes to their attention. Even 
fewer will engage in scientific measurements, 
but all will have ideas about the program’s 
values and weaknesses. 

By being aware of these vested interests and by 
involving representatives of these groups in 
both planning and evaluation, a program direc- 
tor may attract more resources and build a 
stronger program. 

Representatives of such groups will use various 
sets of principles or criteria in making their 
judgments, depending upon their backgrounds. 

Example; An administrator will use princi- 
ples of administration. 

Example; A community leader will use prin- 
ciples of community involvement, 
community development, public re- 
lations, and politics. 

Example: A colleague will evaluate the qual- 
ity, quantity and organization of 
the selected content according to 
the nature of the discipline in- 
volved. 

Example: An educator will pass judgments 
based upon principles and prac- 
tices of education. 

Example: The program director and faculty 
measure the program’s success 
against its stated goals and their 
own professional satisfaction. 

Example: The funding agency will assess a 
continuing education project on the 
basis of principles of cost effective- 
ness and relevance to the mission 
of the funding agency. 

Example: The trainee will pass judgement on 
the educational experience based 
on principles relative to his career 
development and learning relevant 
to problems encountered in his 
work. 
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Borrowing Expertise 

How then can a program director who is rarely 
expert in more than one or two areas assure 
sound evaluation of a program? 

BY UTILIZING THE EXPERTISE OF OTHERS 
AND INVOLVING REPRESENTATIVES FROM 
RELATED DISCIPLINES FIRST IN THE PLAN- 
NING, AND THEN IN THE EVALUATING 
PROCESS. 



use of the 

accompanying charts 



The three charts in the following section are 
designed to help in developing a workable and 
continuous evaluation process usable from the 
earliest planning stages through final evalua- 
tion and writing of a report. 

The charts illustrate an approach to evaluation 
planning. They are not intended to be detailed 
checklists. Not all items will apply to every pro- 
gram. Program directors may find it worthwhile 
to go over each question and make a decision 
as to whether it is applicable and practical to 
Include in his evaluation process. While each 
question may be worth review, it is expected 
that for any given program only a portion will 
be used. 

There is a separate chart on three of the four 
evaluation areas mentioned in the introduction: 
(1) Community Involvement, (2) Education, (3) 
Administration and Finance. 

The term Community Involvement is used in a 
broad sense to mean not only how the program 
relates to other programs and organizations in 
the community, and how they participate in it, 
but how the program contributes to the solu- 
tion of local, .State and national mental health 
problems. 



A program director may call upon experts on 
the staff or consultants from groups such as 
those discussed above. 

Ir. planning as in evaluating, it is particularly 
helpful to keep in mind the definition of evalu- 
ation as a process of gathering data for pur- 
poses of enlightened decision making. 

(Specific suggestions for utilizing expertise of 
persons in related areas and organizations are 
found in final sections of the guidelines.) 



Each of the above evaluation areas is broken 
down on the following pages from three stand- 
points: (1) principles, (2) planning, data gath- 
ering, decision-making, (3) suggested evalua- 
tion questions. (These may be used as follow- 
up questions but in many instances they are 
also questions to v/hich the review committees 
seek answers prior to making grants.) 

The fourth area mentioned in the introduction — 
that of evaluation from the standpoint of the 
specialized knowledge and skills of the disci- 
pline(s) being taught— is not developed in a 
separate chart. Criteria unique to each special- 
ty area are beyond the scope of these guide- 
lines. Each program should assure relevance 
and soundness of specialized knowledge and 
utilize consultants when the expertise is not 
available on the faculty. Evaluation would 
usually be by expert judgment, standardized 
examinations, critical incident studies and 
increments in demonstrable skills acquired by 
trainees. Some general consideration of the 
quality, quantity and organization of the con- 
tent is covered under Principles of Education, 
Chart 2-A. 
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CHART 7 

Education 

IS THE PROGRAM BASED ON SOUND EDUCATIONAL THEORY & PRACTICE? 
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